Method and system of lens shading color correction using block matching

ABSTRACT

Systems, articles and methods to provide lens shading color correction using block matching are disclosed. Example processor systems disclosed herein are to process at least one cluster of blocks of an image to determine at least one modification parameter, modify the first shade correction data based on the at least one modification parameter to determine second shade correction data, and correct a lens shade effect associated with the image based on the second shade correction data.

RELATED APPLICATIONS

This patent arises from a continuation of U.S. patent application Ser.No. 16/927,581, (now U.S. Pat. No. 11,082,675), which is titled “METHODAND SYSTEM OF LENS SHADING COLOR CORRECTION USING BLOCK MATCHING,” andwas filed on Jul. 13, 2020, which is a continuation of U.S. Patentapplication Ser. No. 16/271,458 (now U.S. Pat. No. 10,715,773), which istitled “METHOD AND SYSTEM OF LENS SHADING COLOR CORRECTION USING BLOCKMATCHING,” and was filed on Feb. 8, 2019, which is a continuation ofU.S. patent application Ser. No. 15/722,184 (now U.S. Pat. No.10,205,924), which is titled “METHOD AND SYSTEM OF LENS SHADING COLORCORRECTION USING BLOCK MATCHING,” and was filed on Oct. 2, 2017, whichis a continuation of U.S. patent application Ser. No. 14/965,820 (nowU.S. Pat. No. 9,781,398), which is titled “METHOD AND SYSTEM OF LENSSHADING COLOR CORRECTION USING BLOCK MATCHING,” and was filed on Dec.10, 2015, which is a continuation of U.S. patent application Ser. No.14/841,626 (now U.S. Pat. No. 9,238,377), which is titled “METHOD ANDSYSTEM OF LENS SHADING COLOR CORRECTION USING BLOCK MATCHING,” and wasfiled on Aug. 31, 2015, which is a continuation of U.S. patentapplication Ser. No. 14/498,271 (now U.S. Pat. No. 9,186,909), which istitled “METHOD AND SYSTEM OF LENS SHADING COLOR CORRECTION USING BLOCKMATCHING” and was filed on Sep. 26, 2014. Priority to U.S. patentapplication Ser. No. 16/927,581, U.S. patent application Ser. No.16/271,458, U.S. patent application Ser. No. 15/722,184, U.S. patentapplication Ser. No. 14/965,820, U.S. patent application Ser. No.14/841,626 and U.S. patent application Ser. No. 14/498,271 is claimed.U.S. patent application Ser. No. 16/271,458, U.S. patent applicationSer. No. 15/722,184, U.S. patent application Ser. No. 14/965,820, U.S.patent application Ser. No. 14/841,626 and U.S. patent application Ser.No. 14/498,271 are hereby incorporated by reference in their respectiveentireties.

BACKGROUND

Electronic devices such as smartphones, tablets, and wearable computersare being miniaturized for the user's convenience, which in turninvolves the miniaturization of electronic components in such devices.This includes digital cameras capable of producing digital images. Whilereducing form factors may increase user convenience, it often does so bysacrificing performance or quality. With respect to digital cameras,slimmer form factors place lenses and filters so close to the camera'ssensors that resulting digital images often have color artifacts anddiscolorations. Thus, better quality digital images are desired.

DESCRIPTION OF THE FIGURES

The material described herein is illustrated by way of example and notby way of limitation in the accompanying figures. For simplicity andclarity of illustration, elements illustrated in the figures are notnecessarily drawn to scale. For example, the dimensions of some elementsmay be exaggerated relative to other elements for clarity. Further,where considered appropriate, reference labels have been repeated amongthe figures to indicate corresponding or analogous elements. In thefigures:

FIG. 1 is an illustration of an image with light attenuation;

FIG. 2 is a chart showing light attenuation of the image;

FIG. 3 is an image with lens shading color artifacts;

FIG. 4 is an image with texture that is difficult to provide correctshading;

FIG. 5 is another image with texture that is difficult to providecorrect shading;

FIG. 6 is a schematic diagram of an image capture device for lensshading color correction;

FIG. 7 is a flow chart showing a lens shading color correction process;

FIG. 8 is a flow chart of a detailed lens shading color correctionprocess;

FIGS. 9A-9B is a flow chart of a block matching process for lens shadingcolor correction;

FIG. 10 is an illustration of an image with a cluster of blocks for lensshading color correction;

FIG. 11 is an illustration of an image with motion vectors to locateblocks for lens shading color correction;

FIG. 12 is a diagram of an operation of an example system describedherein;

FIG. 13 is an illustrative diagram of an example system;

FIG. 14 is an illustrative diagram of another example system; and

FIG. 15 illustrates another example device, all arranged in accordancewith at least some implementations of the present disclosure.

DETAILED DESCRIPTION

One or more implementations are now described with reference to theenclosed figures. While specific configurations and arrangements arediscussed, it should be understood that this is done for illustrativepurposes only. Persons skilled in the relevant art will recognize thatother configurations and arrangements may be employed without departingfrom the spirit and scope of the description. It will be apparent tothose skilled in the relevant art that techniques and/or arrangementsdescribed herein may also be employed in a variety of other systems andapplications other than what is described herein.

While the following description sets forth various implementations thatmay be manifested in architectures such as system-on-a-chip (SoC)architectures for example, implementation of the techniques and/orarrangements described herein are not restricted to particulararchitectures and/or computing systems and may be implemented by anyarchitecture and/or computing system for similar purposes. For instance,various architectures employing, for example, multiple integratedcircuit (IC) chips and/or packages, and/or various computing devicesand/or consumer electronic (CE) devices such as imaging devices, digitalcameras, smart phones, webcams, video cameras, video game panels orconsoles, set top boxes, and so forth, may implement the techniquesand/or arrangements described herein. Further, while the followingdescription may set forth numerous specific details such as logicimplementations, types and interrelationships of system components,logic partitioning/integration choices, and so forth, claimed subjectmatter may be practiced without such specific details. In otherinstances, some material such as, for example, control structures andfull software instruction sequences, may not be shown in detail in ordernot to obscure the material disclosed herein. The material disclosedherein may be implemented in hardware, firmware, software, or anycombination thereof.

The material disclosed herein may also be implemented as instructionsstored on a machine-readable medium or memory, which may be read andexecuted by one or more processors. A machine-readable medium mayinclude any medium and/or mechanism for storing or transmittinginformation in a form readable by a machine (for example, a computingdevice). For example, a machine-readable medium may include read-onlymemory (ROM); random access memory (RAM); magnetic disk storage media;optical storage media; flash memory devices; electrical, optical,acoustical or other forms of propagated signals (e.g., carrier waves,infrared signals, digital signals, and so forth), and others. In anotherform, a non-transitory article, such as a non-transitory computerreadable medium, may be used with any of the examples mentioned above orother examples except that it does not include a transitory signal perse. It does include those elements other than a signal per se that mayhold data temporarily in a “transitory” fashion such as RAM and soforth.

References in the specification to “one implementation”, “animplementation”, “an example implementation”, and so forth, indicatethat the implementation described may include a particular feature,structure, or characteristic, but every implementation may notnecessarily include the particular feature, structure, orcharacteristic. Moreover, such phrases are not necessarily referring tothe same implementation. Further, when a particular feature, structure,or characteristic is described in connection with an implementation, itis submitted that it is within the knowledge of one skilled in the artto affect such feature, structure, or characteristic in connection withother implementations whether or not explicitly described herein.

Systems, articles, and methods to provide lens shading color correctionwith block matching.

A multi-function electronic device that has the ability to captureimages such as a smartphone, mobile phone, tablet, laptop, and so forthwith a mobile camera module may often have a form factor with dimensionssized to easily fit in a palm of an average hand of a user. In suchcases, the small electronic device may have a depth so small it may bemeasured in millimeters (mm). For instance, conventional smartphones mayeasily have a representative depth of 7.6 mm or thinner. When a deviceof such narrow depth incorporates an image capture module, such as adigital camera, it necessarily decreases a distance between cameraoptics and an image sensor. The decreased distance causes light tostrike various portions of the image sensor in a non-uniform manner,such as concentrating light towards a center of the image sensor andattenuating light towards the outer edges of the image sensor as shownon the image 10 (FIG. 1 ) produced with a sensor and using a flat testarea. As shown in FIG. 1 , corner regions 12 of image 10 are darker thana central region 14 of image 10. This suggests that light attenuationdue to optical lens shading is greater with respect to corner regions 12than it is with respect to central region 14. This attenuation may notbe static throughout color planes which results in severe colordistortions. Typical results are digital images that are brighter in thecenter and darker near the edges as mentioned, as well as artificialcolors, and attenuation with respect to the original scene. FIG. 2 showsone color plane from FIG. 1 where the vertical axis is brightness, andthe base shows pixel coordinates. The light is brightest in the centerand reduces significantly toward the outer corners. FIG. 3 shows thatunequal light attenuation, and respectively color distortions, evenoccur in high end devices such as a smartphone. Distortions 302 areshown on the image 300 in attempts to correct the color as well as colorartifact area (304) (which may be tinted too red in this example) thatremains despite such corrections. Thus, a problem still exists withimage quality of even well-known top-end devices as shown in FIG. 3 .

More specifically, since a majority of digital imaging sensors areequipped with CFAs (color filter arrays) to reproduce color informationin digital images, as a result, such attenuation varies in differentcolor channels, which results in severe color degradation when the imageis reconstructed improperly. This attenuation varies depending on lightspectrum, module design (geometry and optics), IR-cut-off (infra-red)filter, sensor design, and potentially other factors. For one factor,when the optics (or filter) and sensor plane are brought closer togetherin these small devices, this creates a relatively sharp chief ray angle(CRA) relative to the sensor plane, and depending on the design of thepart that holds the sensor, not as many light rays compared to someideal arrangement may have a clear path to the sensor. Since suchdistortion can be measured objectively and it is visible to users, theremoval of these distortions is an important criterion for imagequality.

To correct for such non-uniformity, an image capture device mayimplement various lens shading correction (LSC) algorithms tocompensate. An LSC algorithm may utilize one or more correction LSCtables with correlated color temperature (CCT) factors associated withthem to perform such correction. Once an LSC table is formed (or moreaccurately selected among pre-stored LSC tables), it is applied topre-processed image data to make the corrections before the data isprovided to an encoder or before the image is displayed for example. Theconventional correction processes using LSC tables, however, still aretoo inaccurate. They may cause unequal attenuation, and in turn visibledistortions. Previous solutions were based on use of hue or saturationhistograms (global statistics analysis). A better solution is based onlocal color deviation analysis (local statistics analysis) which isdisclosed in U.S. patent application Ser. No. 14/182,284, and isexplained in greater detail below.

One of the principle challenges in solving lens shading problems duringcamera run-time is in understanding which part of an image can beutilized best in calculation of color distortion metrics. Specifically,the more accurately a distortion (such as distortions 302) can beidentified and measured, the better chances the correct LSC table can beselected to properly compensate for the distortion. Color distortionsare typically measured by a mean square error (MSE) with respect toorigin in a hue, saturation and value (HSV) color space projected to ahue/saturation plane in a block of a corrected and white-balanced image.It has been found that the distortions are more easily identified andmeasured in flat, achromatic areas of the image (also referred to assmooth areas). Thus, the simplest and most efficient way to finddistortions is in a flat-field image. Such an image may be acquired witheither diffusive glass or an integrating sphere. In such a case, thesolution is rather straightforward. Correction is performed by revertingthe observation itself. In other words, some preferred color orbrightness coefficient is determined, and the darker areas are madelighter and the lighter areas are made darker by the coefficient.However, such an approach only may be feasible in a test laboratoryenvironment, and not applicable when correcting typical images that mayor may not have wide brightness and color variations.

Hence, a search may be performed for smooth or uniform areas on typicalimages and analyzed during run-time to enable calculation of distortionmetrics. Such calculations require both sufficient data and propertiesdescribed above to make a reliable decision about the LSC tableselection. One solution to this problem was to divide the image into anumber of rectangular blocks, such as 24 blocks where each block is arelatively large continuous area of the image. An error is calculatedfor each block based on hue and saturation value, each illuminant (or inother words, light source), and the size of the block. This error isthen used to calculate a weight associated with LSC tables as describedbelow. Important here, only some of the blocks are selected for use toultimately select LSC tables even though errors are calculated for allof the blocks in the image. By one approach, a select number of theblocks with the lowest error(s) are then used to select the LSC tables.In the previous approach provided by U.S. patent application Ser. No.14/182,842, two to eight of the 24 blocks with the lowest error valuesare used for further calculations. This block selection operation wasused because the smaller the error, the more likely the block provides asmooth portion of the image, or at least more smooth sections within theblock. It was assumed that the blocks with the largest errorsrepresented areas of an image with great variations in saturation andhue suggesting many edges between objects depicted in the block, andtherefore, the block does not show a smooth area of the image.

This previous approach of block selection, however, still has problemswith images that have complex textures where the smooth areas are notconcentrated in one relatively large section of the image. For example,images with small repetitive textures are some of the most difficult foranalysis. This includes image content such as tables or spread sheets onimage 400 (FIG. 4 ) and with lines that divide rows and columns to formalternating patterns of dark and light regions on the image. Likewise, acheckered fabric pattern on an image 500 (FIG. 5 ) forms alternatingpatterns of white and color sections. The previous operations that uselarge blocks for analysis may not clearly have one block that representsa smooth area of the image, and in turn, the LSC table selection canbecome inadequate. In addition, the conventional segmentation solutions(for example, flood fill segmentation) that are limited to a continuousarea of the image may provide inefficient or insufficient results, ormay fail to provide any results, because an insufficient portion of thedata of the smooth areas of the image are included in the selectedsegmented area or selected blocks.

To solve these problems, the approach disclosed herein uses a dynamicsearch in one or more video frames in a video sequence to form at leastone cluster of blocks for a frame. The cluster of blocks represents asmooth uniform area of the image. A smooth uniform area refers to anarea of the image with the smallest amount of edges, or in other words,without large jumps in luminance or color ratios. This may provide amuch larger amount of image data for each cluster compared to theconventional methods. The blocks or areas that form the cluster are notnecessarily connected, and may be included in the cluster regardless ofwhere the block is located on the image. Thus, the blocks in the clusterare not necessarily in contact with each other, and the cluster need nothave any certain shape.

The disclosed image modeling process includes selecting blocks for thecluster (also referred to herein as self-similarity block matching) whenblocks have the same property such that the data of both blocks pass oneor more criteria related to the image data. By one form, a referencearea or block in the image (or more accurately in a statistics grid ofimage data) is compared to multiple blocks in the image. The statisticsgrid may be a grid of luminance values or color values or color ratiosthat are spaced on the grid to correspond to the position of blocksdefined on the image. When the comparison satisfies similaritycriterion, the two blocks are said to be of the same object, or of thesame properties, and the found block is included in the cluster. Thecluster of blocks is then used to find errors in hue and saturation forexample, which in turn are used to find weights for the LSC tables.Selected ones of the weighted LSC tables are then provided to correctpre-processed images.

The approach described here may be referred to as a non-local solutionsince it incorporates benefits of both a global (histograms basedsolution) and local statistics analysis (by performing color distortioncalculations described further below). Since the clustering arrangementuses more efficient data utilization compared to the conventional lensshading color correction methods, the quality of classification,run-time robustness, and increased scope of applicability are improveddespite an increase in cost of computational complexity.

Referring to FIG. 6 , an image capture device or system 600 to processcaptured images and apply the lens shading color correction withself-similarity block searching is described. The image capture device600 may be a stand-alone device, such as a digital camera, or may merelybe a camera module on a device such as a smartphone. In the presentexample, the image capture device 600 may have a camera module 606 thathas optics 602 and one or more image sensors 604, where each optic ispaired with a corresponding image sensor 604. Each of the image sensors604 may be based on any of a variety of technologies for capturing animage of a scene, including and not limited to charge-coupled device(CCD) semiconductor technology. Each of the optics 602 is made up of oneor more lenses, mirrors, prisms, shutters, filters, etc. employed toconvey images of a scene and to at least partly define the field of viewof corresponding ones of the image sensors 604. The image sensors 604and the optics 602 (whatever their quantity) are positioned and orientedrelative to each other in a manner intended to provide each image sensorand optics pair with a field of view that overlaps field(s) of view ofone or more of the other image sensor and optics pairs.

The image capture device 600 also may have a pre-processing unit 636that receives raw image from the camera module 606 and performsblack-level correction on the raw image data, upscales a LSC correctiongrid (or table) received from a lens shading estimation unit 618described below, and upscales the grid to the size of image data, andmultiplied in a point-wise manner.

A downscaler and statistics (DS) unit 608 will then use the LSCcorrected image data to generate statistics for 3A, which comprises LSCcorrected, AF statistics for auto-focus, and black-level corrected rawstatistics. Thus, the DS unit 608 may provide both the LSC correcteddata for 3A corrections as well as block matching, and the black-levelcorrected raw data to the lens shading estimation unit 618 forapplication of candidate LSC tables and lens shading estimations. Insystems that do not provide the black-level-adjusted data to the 3Aunit, the LSC corrected data from the downscaler may be reverted beforeit is used in lens shading estimation. The reversion may be performed bythe downscaler or a separate module for such purpose. The DS unit 608also is provided to downscale the raw data stream (or raw imageinformation) from the camera module 606. The DS unit 608 also analyzesthe downscaled image data to form statistics that may be used by the 3Aunit 610. At the very least, the pre-processed image data should alreadybe processed for black-level correction before lens shading estimationoperations may begin. Lens shading estimation unit 618 expectsstatistics to be black-level corrected only. Hence, if the DS unit 608assumes only one type of statistics as an output, which isLSC-corrected, then this correction shall be reverted and used as inputto system 618 as mentioned above. Other 3A units may use LSC-correctedor AF statistics, or both. The preprocessed image may be a single image,or a sequence of images, such as from a video sequence. The preprocessedimage may be received by the LSC estimation unit 618 in real-time, or innon-real time from a memory unit. Implementations are not limited inthis context. Note that pre-processing and post-processing discussedherein are relative to the LSC shading estimation.

The 3A unit 610 has an automatic focus (AF) unit 612 to perform AFcalculations to determine factors to move a camera lens into focus. Anautomatic white balance (AWB) unit 614 is provided to performcalculations to image data for white balance, and an automatic exposure(AE) unit 616 to provide factors to adjust exposure time for capturingan image. The 3A unit 610 also may have the lens shading estimationsystem or unit 618.

The LSC estimation unit 618 may use a correlated color temperature (CCT)estimate unit 620 to determine which of a set of LSC tables 622 are tobe used for lens shading color correction for a current image.Specifically, a characterization phase is performed during design andassembly of the image capture device 600 before a correction phase. Inthe characterization phase, LSC tables 622 are created to cover most CCTvalue ranges for commonly used light illuminants. The LSC tables 622 arestored in a memory, such as a persistent storage, for later use duringthe estimation and correction phases. Role of system 618 is to decidewhen, and in which proportion, each LSC table 622 should be used tocompensate shading of a digital image.

A correction phase may be performed in real-time, or run-time executionduring use of the image capture device 600 and when data is streamedfrom the camera module 606. During the correction phase, the 3A unit 610via the DS unit 608 may analyze the content from the streamed imagedata, and supplementary metadata. Examples of metadata may include,without limitation, auto-focus values, CCT estimate, time values, and soforth. The relevant data is provided to the 3A unit 610 by the DS unit608. During the correction phase, the CCT estimate unit 620 is used toestimate one or more original light source types and initial selectionof candidate LSC tables. The details for selecting candidate LSC tablesis explained below.

The LSC estimation unit 618 also may include a block selection unit 626.The block selection unit 626 also may be referred to as a block matchingunit, segmentation unit, or clustering unit as well. It will also beappreciated that the device 600 may have at least one LSC unit toperform both the operations of the LSC estimation unit 618 and blockselection unit 626, such that both units 618 and 626 may be consideredas part of one LSC correction unit, or any other variation. The blockselection unit 626 may receive AF statistics, and provides approximationof flat smooth areas in the scene by using the AF statistics. The blockselection unit 626 may output definitions of block clusters thatrepresent the smooth areas. This may include a total area of the cluster(total number of pixels in the cluster) as well as the coordinates ofthe data in the cluster on an image. The coordinates may be used toobtain the hue and saturation values for the data in the cluster. Thecluster size, and hue and saturation values, may be used to determineerror values as explained below. Each image may have one or moreclusters. The total amount of pixels from all of the clusters in animage should contain a sufficient amount of image data to provide arobust estimate of deviation between color samples (for example,recommended minimum of 100 pixels or other relevant value). The detailsfor generating the clusters are provided below.

An initial (or candidate) LSC table unit 624 may be provided to identifythe tables initially selected by the CCT estimate unit 620, and forfurther processing. Each set of the initial LSC tables for Light₁, . . ., Light_(L) (unit 624) as well as the image data of the blocks in theestablished clusters of an image may be provided to a grey-worldwhite-balance (GW-WB) unit 627, which may be controlled by the AWB unit614. The unit 627 applies white-balance calculations to the LSCcorrected statistics of at least the clusters in the image to adjust theimage data for white balancing in the clusters. A red/green/blue (RGB)to hue/saturation/value (HSV) convertor unit 628 is provided to convertthe RGB values of the clusters in the image to HSV color space valuesfor error calculations. An error unit 630 then uses the HSV clusterimage data (such as the hue and saturation values of the blocks in thecluster) as well as cluster sizes (or totaled data in the clusters) tocalculate an error. For each image cluster, the GW-WB module 627, RGB toHSV convertor 628, and error unit 630 processes cluster of data of aninput statistics, and outputs an error value for each cluster (and foreach illuminant). Separately, a probability and confidence unit 632 alsoreceives the cluster data from the block selection unit, and calculatesa confidence value for each cluster by one example. The error unit 630then uses the error values and corresponding confidence values togenerate a weight (wl to wL) for each initial (or candidate) LSC tableassigned to each illuminant. A fusion unit 634 then adjusts the initialLSC table data by using the weight for the table, and then summing (orfusing) all of the LSC tables from all of the illuminants into a singlecomposite LSC table. The fusion unit 634 may then output one final LSCtable to the pre-processing unit 636 to adjust the pre-processed imagedata for lens shading color correction.

The pre-processed, LSC corrected image then may be provided, via the DSunit, to post-LSC processing 637 that may perform tasks such as noisereduction, pixel linearization, resolution reduction, Bayer demosaic,and/or vignette elimination as well as other tasks, and an encoder 638,which may compress the data for transmission in a bit stream or forstorage on an image readable media.

By one option, the encoder 638 may have an inter-prediction unit 640that generates motion vectors (MVs) to determine the motion of blocksfrom frame to frame in a video sequence of frames forming the images. Inthis case, the motion vectors, or the new locations for blocks insubsequent frames, may be provided to the block selection unit 626 sothat the computational load for statistics is reduced. Also, a fullsearch of the entire frame for clusters does not need to be performedfor every frame (the term frame is used interchangeably with the termstatistics herein). Thus, the block matching or selection unit 626 mayonly be used on every nth frame (every 10*, or 20^(th) for somenon-limiting examples), or may be provided only on certain types offrames, such as only on P-frames (predicted frames—often from oneprevious frame), but by one example not on B-frames (predicted from bothforward and back along a video sequence) or any other type of frame. Byone approach, as explained below, motion vectors may be used to reducethe computational load for the search for at least some of the blocks inan image. In this case, the block searching may be limited to thoseimages along a video sequence used for inter-prediction by an encoder.In other words, the images used by the encoder to determine the motionof a block from image to image. Many other examples are contemplated aslong as the cost of searching for clusters on a frame is reduced. Thus,it may be possible to only search for a certain number of clusters on animage (such as one to three) and use MVs for the remaining clusters inthe image, and so forth. For instance, there may only be an update onknown clusters with MVs, or only update say 1-3 clusters out of a totalof 6 clusters with MVs that represent frame the best (say those thatcontain largest amount of data), without carrying out a search procedureat all. In other words, the system applies MVs and does not search fornew clusters in the next 5, 10, 20, or other interval of frames.

Referring now to FIG. 7 , by one approach an example process 700 is acomputer-implemented method of lens shading color correction with blockmatching that may be used to correct image data. In the illustratedimplementation, process 700 may include one or more operations,functions or actions as illustrated by one or more of operations 702 to712 numbered evenly. By way of non-limiting example, process 700 may bedescribed herein with reference to example image capture devices 600 and1300 of FIGS. 6 and 13 respectively, and where relevant.

Process 700 may include “obtain at least one statistics grid of pixeldata spatially corresponding to a frame having blocks” 702. This process700 also may comprise “select at least some of the blocks to be includedin at least one cluster of blocks” 704. To assign blocks for thecluster, the process 700 may include “choose candidate blocks at leastwith a difference in luminance or color ratio between the candidateblock and a reference block that meets a criteria” 706. As explainedbelow, it does not matter where the blocks are located on the image, andwhether or not the blocks are in contact with other blocks in thecluster. Also, by one example, the criteria may be one or morethresholds such as a Euclidean distance between block values as well asother criteria described below. The process 700 also may include “findan error associated with the color of the blocks in the cluster” 708.This error, also explained below, may be associated with the hue andsaturation values within the blocks in the cluster as well as the areaof the cluster. Process 700 may also include “use the errors todetermine a weight for one or more LSC tables” 710 as well as “form acomposite LSC table based on the weights and to be applied to correctthe color at the frame” 712.

Referring now to FIG. 8 , a more detailed lens shading color correctionprocess 800 with block matching described herein is arranged inaccordance with at least some implementations of the present disclosure.In the illustrated implementation, process 800 may include one or moreoperations, functions or actions as illustrated by one or more ofoperations 802 to 826 numbered evenly. By way of non-limiting example,process 800 will be described herein with reference to example imagecapture devices 600 and 1300 of FIGS. 6 and 13 respectively whererelevant.

Process 800 may include “obtain image data” 802, and as explained abovemay include obtaining raw image data streamed from a camera module orother image capture module. The camera module, such as camera module606, may be part of a single image capture device that includes thedisclosed lens shading color correction system, or it may be remote fromthe disclosed system.

Process 800 may include “initiate pre-processing” 804. This may includeblack-level and lens shading corrections. The resolution reduction, AWBoperations, RGB to HSV space conversion, and the lens shading colorcorrection itself also may or may not be considered part ofpre-processing by some examples. Where these operations are consideredto be part of pre-processing, their operation may or may not beperformed by a pre-processing control. As mentioned above, by oneapproach, only black-level correction is performed on an image beinganalyzed before the following lens shading color correction operationstake place with regard to that image.

Process 800 may then include “downscale data” 806 to reduce the amountof image data to a uniform, controllable amount, such as a grid of data(corresponding to pixel locations) to form a 39×51 image for asmartphone for example. Such downscaling may be performed by usingstandard sampling techniques.

Process 800 may include “calculate CCT estimate” 808. As mentionedabove, pre-processed image content may be obtained, and a set of initialor candidate LSC correction tables may be selected 810 for correcting acurrent image being analyzed. The candidate LSC tables may be selectedby first analyzing image content and by calculating a probability CCT(PCCT) of presence of illuminant from a characterization database usingsub-sampled input image data. To state it another way, in someimplementations, a respective probability of presence may be calculatedfor each of a plurality of LSC tables, and the set of candidate LSCtables to be used going forward may comprise each LSC table among theplurality for which the probability of presence is greater than zero. Invarious such implementations, each chosen LSC table may correspond to adifferent respective illuminant, and the probability of presence foreach LSC table may indicate a probability of presence of the illuminantto which that LSC table corresponds. It will be understood that manydifferent variations or alternatives for the candidate LSC correctiontables are contemplated. The selected set of LSC correction tables arethen to be held or identified by initial LSC unit 624, and then weightedas described below for application to a current preprocessed image.

Separately from the selection of the candidate correction tables, blocksof the image data may be included in one or more clusters that representsmooth areas of the image, and are to be used in calculations goingforward to generate a composite LSC correction table to be applied tothe pre-processed images before the data is used for display or sent toan encoder for example. Thus, process 800 may include an operation of“match blocks to form clusters” 812. Process 900 described below is onepossible example methodology for the matching of the blocks andgeneration of clusters. The output from this block matching process 900may be data that includes the cluster sizes as well as the location ofthe pixels in each of the clusters. This cluster data then may be usedto determine weights for the LSC tables as described below.

Referring to FIG. 9 to explain the block matching and generation ofclusters in more detail, example process 900 is provided for selectingblocks to be included in a cluster to use the clusters, rather than anentire image, to perform lens shading color correction. This isaccomplished by matching blocks for placement in clusters as describedherein to provide a relatively high quality image. Process 900 isarranged in accordance with at least some implementations of the presentdisclosure. In the illustrated implementation, process 900 may includeone or more operations, functions or actions as illustrated by one ormore of operations 902 to 954 numbered evenly. By way of non-limitingexample, process 900 will be described herein with reference to exampleimage capture devices 600 and 1300 of FIGS. 6 and 13 respectively whererelevant.

Process 900 may include “obtain image data” 902, where the at leastpartially pre-processed image data is provided to the block selectionunit 626 for example. For this operation, let:={x=(x ₁ ,x ₂): x ₁=1, . . . ,N,x ₂=1, . . . ,M}  (1)be the set of coordinates where a statistics grid (and correction grid)is defined, and where each point on the statistics grid corresponds to apixel of a downscaled image or frame of a video sequence being analyzed.Here, x₁ is the horizontal coordinate, while x₂ is the verticalcoordinate on an image or grid of size (N, M). The statistics may beBayer data, and comprises at least four-scaled Bayer color channels orelements {y_(R), y_(G1), y_(G2), y_(B)} associated with each point onthe statistics grid, and in turn a single image pixel on a frame. By oneexample, the Bayer color elements are used to form a luminance channely(x) for block matching analysis, while in other examples color ratiosformed by using the Bayer color elements may be used for the blockmatching instead as explained below. If the sensor on the image capturedevice being used comprises other than Bayer layout or color filters,then correction grids may be adjusted accordingly and the process belowstill may apply.

When luminance values y(x) will be used instead of, or in addition to,the color ratios to determine block matching, process 900 also mayinclude “obtain AF statistics” 904 when the AF statistics may be used todetermine whether similarity conditions are met for inclusion of blocksin a cluster, and may include derivative values as explained below.

Referring to FIG. 10 , an exemplary image (or statistics grid of imagedata such as luminance or color values) has a group A of pixels and aseparate group B of pixels that may be placed together (union operation)in a single cluster or ROI or Xs. As mentioned above, each statisticsgrid may be divided into blocks that are 3×3 (or 4×4) pixels or pointsor other sizes that may or may not be consistent with block sizes usedby an encoder analyzing the video sequence. Going forward the locationof points on the statistics grid and corresponding pixel points (orimage pixels) may be referred to interchangeability herein forsimplicity. Each smooth area or data cluster (or just cluster) of pixelswith similar properties found in the image and that is to be treated asa single cluster is called a region of interest (ROI) X_(s). The unionof blocks in area A and B on image (frame k) 1000 collectively are asingle cluster or ROI X_(s), The image, however, may have more than oneROI X_(s). Thus, s=1 to S where S is the count of the number of clustersor the number of ROIs within an image. In other words, S may be muchsmaller than the total count of points in the statistics grid (S<<MN).Each ROI may be referred to as an ROI X_(s), while the total number ofROIs or clusters in an image is S.

Referring to FIG. 9 , process 900 then includes a check 906 whether aprevious image has already been analyzed and has a defined number ofclusters. The check operation 906 is performed to determine whethermotion may be used to reduce the computational load of the bockmatching. Specifically, since block matching and selection of blocks toplace in clusters described herein is computationally extensive,especially when executing the block matching for each frame in a videosequence, a video encoding engine or video stabilization engine(encoder) can be used to provide motion vectors (MVs or Ax) to improveat least the initial approximation of the location of a region ofinterest (ROI) for a new frame k+1 as explained below.

Thus, this operation or check tests whether or not a previous imagealready has an established total S that would be relevant to the currentimage being analyzed. In some forms, the previous image may be an imagek while the present image being analyzed is an image k+1. The previousimage with S ROIs may be the adjacent previous image relative to thecurrent image, or may be spaced along a video sequence a certain numberof images away (or divided by certain types of images as alreadyexplained above). The spacing between images used for lens shading colorcorrection may not be limited except that the previous image should havecontent that is still relevant to the current image being analyzed whenmotion data will be used to reduce the computational load of the blockmatching as explained below. Thus, in most cases, the previous imageshould be part of the same scene as the current image and within thevideo sequence being analyzed.

If there is no previous relevant image, such as for the first image in avideo sequence or scene, then S is set to one (908), and the processcontinues without the use of motion data at operation 924 but may beimplanted to analyze the subsequent images. Also, optionally, theprocess 900 may continue at operation 924 omitting the motion-basedoperations, when use of motion data is to be completely omitted for anentire video sequence or scene, for example.

When there is a previous image with an established S ROIs, and motiondata will be used, then process 900 may include “set current S same as Sfor previous image” 910. Another way to say this is to set current imageS_(k+1)=previous image S_(k).

Process 900 may then include “obtain motion data for next image (k+1)”912. Particularly, the images may be divided into blocks, such as 3×3 or4×4 pixel blocks or other sizes. An encoder processing the videosequence with the current image may be performing inter-prediction bygenerating motion vectors that indicate the motion of the blocks fromone image to another image. Referring to FIG. 11 , in the presentexample, the motion vectors must provide the motion of the blocks fromthe previous image that has the established S to the current image inorder to form the initial S on the current image being analyzed. Also,by one form, the motion vectors (value that indicates change in position(Ax)) may be provided to the block selection unit 626 to calculate thecurrent (or new) coordinates of the blocks in the current image, orotherwise the current block coordinates in the new image may becalculated by applying the motion vectors by the encoder for example,and the new coordinates are then provided to the block selection unit626 instead of the vectors. As shown on image 1100, motion vectors Δx(A)and Δx(B) show the motion of block groups A and B from frame k to framek+1.

Process 900 may then include “update location of ROIs Xs from previousframe according to motion” 914, and specifically, according to motionvectors of blocks or new motion coordinates of blocks obtained byapplying the motion vectors. For instance, there may be a memory orbuffer for the current image with a list or index that lists the currentROI found in the current image and providing the block locations thatare assigned to each ROI Xs, thereby also indicating which image data(luminance, and color values) apply to those locations and areconsidered part of that ROI. Thus, the application of the motion vectorsto find the new locations of the ROI Xs may be understood as:X _(s) ^((k+1)) ={x+Δx:x∈X _(s) ^((k))),Δ∈ΔX ^((k+1))}  (2)where image or frame k is the previous image, k+1 is the current image,X_(s) for k-th iteration is denoted by X_(s) ^((k)), and x is thehorizontal and vertical coordinates (x1, x2) on a statistics grid orimage, and that represents the location of a block (such as the upperleft corner of a block) on a map of motion vectors ΔX^((k+1)), and byone example, is assumed to be of the same size as a size of inputstatistics, where:ΔX ^((k+1)) ={Δx=(Δ(x ₁),Δ(x ₂): x ₁=1, . . . ,N,x ₂=1, . . . ,M}  (3)where N, M are the size (and final coordinates) of the statistics grid(or image). Once the clusters or ROIXs carried from the previous imageare located on the new image, those ROI are maintained on the index orlist of ROI for the current image.

Process 900 may then include “remove ROIs Xs not meeting criteria” 916.This includes removal of ROI that were completely eliminated from theprevious image and did not carry over to the new image. Additionally, byone example, an ROI carried over from the previous image by motion datamay only be set when the ROI meets minimum criteria. This may include aminimum size of the ROI such as a minimum number of blocks or minimumnumber of pixels (note that this may be regardless of the position ofthe block or pixel in the image as long as the other luminance or colorvalue criteria below are met). Thus, by one example, the ROI is droppedfor the current image if.#(X _(s)∩

(x _(ref)))<#

(x _(ref)),  (4)such that the total number of elements (#) (for example, the number ofgrid points or pixels) in the cluster or ROI Xs not including thereference block B(x_(ref)) is less than the number of elements (#) inthe reference block B(x_(ref)) itself, then the ROI Xs has insufficientdata to provide a significant contribution to the smooth area of theimage to determine table weights. In these cases, process 900 may theninclude “decrease S for each Xs removed” 918 so that the total amount ofROIs S can be reduced, and the insufficient or missing Xs is thendropped from the index or list of ROI for the current frame.

Process 900 may comprise “refine block matching for reference data frommotion Xs” 920. Particular, as mentioned, the x_(ref) and Xs found byusing the motion vectors (and/or all Xs defined for the image) may beconsidered as preliminary or initial locations. Thus, by one example,block matching calculations such as those described below are stillcarried out for the Xs for the current image established with motiondata to make a more sure determination that the clusters or ROI arevalid. Thus, the motion-based B(x_(ref)) is compared to each block B(x)in the image to determine if it meets the matching criteria describedbelow. By one example, those blocks that do not meet the block matchingcriteria are discarded, and those that do meet the criteria area added.Once all available candidate blocks in the image are tested, the ROIstill may be eliminated if it does not pass the size criteria mentionedabove. If the ROI is still sufficiently large, it may remain on the ROIlist for the current image except as redefined by the refinementprocess. By other examples, only those blocks in the motion-based ROImay be tested with the established motion-based B(x_(ref)) to defineblocks to be taken out of, and blocks to remain in, the ROI. Many otherexamples are possible. By one alternative option, refinement blockmatching is omitted, and the established, carried-over ROI are assumedto be valid for table weight calculations without refinement. Theidentification of the ROI Xs established using the motion data is thensaved, and specifically process 900 may comprise “add Xs from motion tolist of current ROI Xs” 922.

The process 900 then may continue with operation 924 to establish newROI for areas of the current image not already assigned to themotion-based ROI Xs. Particularly, for the present example, thereference point X_(ref) defines the upper left corner of the referenceblock B(x_(ref)), and the grid or image point x defines the upper-leftcorner of the candidate blocks B(x) that are to be tested for a matchwith the reference block.

By one approach both the reference block and the candidate blocks are inareas of the image not already in a cluster or ROI. One way to statethat the reference block used to establish new ROI (also referred to asclustering) in an area not already within the Xs established by themotion data is to state that:

(x _(ref))∈X−U _(i=1) ^(S) X _(s)  (5)or in other words, block matching for reference data is from the set X−UX_(s) (NOT the union of all ROI X_(s) already established). Forrefinement (operation 920) block matching is from a search for blocksB(x) from each set X_(s) ^((k+1)) s=1, . . . , S executing blockmatching in X−U_(i=1) ^(S)X_(s) only, i.e., where:

(x)∈X−U _(i=1) ^(S) X _(s) and

(x _(ref))∈X _(s)  (6)

Thus, process 900 may include “Set x1=1, and x2=1” 924 to test the firstlocation on the grid or image to determine whether the reference pointX_(ref) or (x_(1,ref), x_(2,ref)) can be located at (1, 1). A check isthen performed to determine whether point (x1, x2) is already locatedwithin any ROI 926. If not, reference block (or specifically itsupper-left corner) B(x_(ref)) or (x_(1,ref), x_(2,ref)) is set at (x1,x2), and the block matching process proceeds.

If (x1, x2) is already in a ROI, then process 900 performs a routine tocheck another point on the grid or image to be the reference point(x_(1,ref), x_(2,ref)) (or anchor for the reference block B(x_(ref))).By one example, the reference point is searched and set in rasterfashion starting from the upper-left of the image or grid and proceedingdown the row to the right and starting at the left of the next lowerrow, and repeating until the entire image is searched. For example, ablock size (for a reference block B(x_(ref))) may be (K1, K2) in pixelsor grid points, and as mentioned above, the image size may be (N, M).The routine may include testing whether x1>N−K₁, or in other wordswhether the last block position along a row has been reached. If not, x1is set to x1+1 (930), and (x1, x2) is retested for prior inclusion in anROI (928). If the end of the row has been reached, x2 is set to thebeginning of the next lower row (x2=x2+1 and x1=1) (932). This newposition of x2 is then tested (x2>M−K₂) (934) to determine whether thelast (or lower-most) row for a block position has been reached. If not,the new point (x1, x2) is retested for prior inclusion in an ROI. If ithas been reached, then all of the possible reference blocks B(x_(ref))already have been found, the block matching process is ended, and theprocess 900 continues with lens shading color correction process 800(operation 814). It will be understood that many variations for settingB(x_(ref)) may be used, and by one example approach, the points orpixels around (x1, x2) that would be included in the reference blockB(x_(ref)) also must be outside of any ROI for the reference block to beset at that location.

Once a point (x1, x2) that is not within an ROI has been found,B(x_(ref)) is set at (x1, x2), or particularly the upper-left corner isset at (x1, x2) (936). A similar operation then may be performed to finda next B(x) not already in any ROI Xs (938) and to be included in thecluster of the B(x_(ref)). Similar to the search for the location of thereference block, the search for the candidate block may be performed inraster fashion from left to right along a row, and then downward row byrow in the grid or image as well. Once each candidate block B(x) isfound, or when all of the possible B(x) candidates are found in theentire image, the candidate blocks may be compared (940) to thereference block to determine whether a match exists.

The problem may be stated as to find all such clusters of data X_(s)where X contains a sufficient amount of data to analyze and make arobust decision about which of predefined shading correction tablesT_(l), l=1, . . . , L provide the smallest color distortion in X_(s)where l is the count of illuminants (light sources) detected in theimage. For the sake of simplicity, shading correction tableT_(l)(x)={T_(l) ^(Gr)(x), T_(l) ^(R)(x), T_(l) ^(B)(x), T_(l) ^(Gb)(x)},where x∈X, may be a combination of four correction grids, with one foreach of the Bayer color channels.

By one option, the comparison may be based on luminance values formingthe statistics grid mentioned above. To perform the matchingcalculations, let:

(x _(ref))={=x=(x ₁ ,x2): x ₁ =x _(1,ref) , . . . ,x _(1,ref) +K ₁ ,x ₂=x _(2,ref) , . . . ,x _(2,ref) +K ₂}  (7)where x_(ref)=(x_(1,ref), x_(2,ref)) is the reference point, and K₁+1(K₁≥0) and K₂+1 (K₂≥0) are the dimensions of the block of coordinates

(⋅), and

(x_(ref)) is the reference block coordinates. Generally, two blocks Y (

(X_(ref))), and Y(

(x)) are called to be similar if the distance d(Y(

(x_(ref))), Y(

(x))) between these two blocks does not exceed threshold T:d(Y(

(x _(ref))),Y(

(x)))≤τ,  (8)where Y(

(x)) refers to a set, i.e., Y(

(x))={y(x):x∈

(x)}, while y(x) is a scalar value of luminance. Distance d can becalculated as:l _(p) metric d _(p)(α,β)=(Σ_(i=1) ^(n)|α_(i)−β_(i)|^(p))^(1/p)  (9)while using l₂ (i.e. p=2). Hence, for each point x_(ref), X_(s) can bedefined as a union of at least two rectangular blocks of data, which arenot necessarily connected (e.g., adjacent on the grid or image):

$\begin{matrix}{{X_{s} = {{{\mathcal{B}\left( x_{ref} \right)}u}\bigcup{\mathcal{B}(x)}}},} & (10)\end{matrix}$Thus, by this alternative, the similarity between blocks is determinedby using y(x) luminance values. In one example form, the similaritybetween the blocks, and in turn the inclusion of the block in a cluster,is established when all of the following conditions are satisfied, andwhich will be discussed in turn:

$\;\left\{ {{\begin{matrix}{{{x_{ref}\epsilon\; X} - {\bigcup_{i = 1}^{s - 1}X_{i}}},} & (11) \\{{d\left( {{Y\left( {\mathcal{B}\left( x_{ref} \right)} \right)},{Y\left( {\mathcal{B}(x)} \right)}} \right)} \leq \tau} & (12) \\{{{{Y^{\prime}\left( {\mathcal{B}\left( x_{ref} \right)} \right)}} \leq \tau^{1}},} & (13) \\{{{{Y^{\prime}\left( {\mathcal{B}(x)} \right)}} \leq \tau^{1}},} & (14)\end{matrix}{where}\mspace{14mu} x} \neq {x_{ref}.}} \right.$

Regarding equation (11), the first criteria, as mentioned above, is thatat least the reference point, and by one approach the entire referenceblock anchored at that point, cannot already be located in a previouslyestablished cluster or ROI Xs.

Regarding equation (12), this is the Euclidean distance from eq. (9)above between the block at the reference point B(x_(ref)) and thecandidate block B(x) being analyzed for inclusion in the currentcluster. If the distance is smaller than the threshold, then thecandidate still may be included, and the next equation is checked. Thethreshold is set during system simulation and testing to achieve bestperformance. It shall be larger than 0 and lower than max value of y(x),or y′(x) respectively, normalized with respect to block B(x) size. Sayfor τ, it is assumed that difference equal to 10 between pixels isminor, and two blocks of data containing those pixels are ‘similar’.Than a sum of squares of K elements of each pixels of value 10 issqrt(sum(all K value of 10{circumflex over ( )}2))=sqrt(K*10{circumflexover ( )}2)=10*sgrt(K) which is the threshold. The threshold is adjustedempirically until the best system performance is achieved.

Regarding both equations (13) and (14), one assumption imposed on y(x)is that it is locally smooth, meaning the function is differentiable ineach point x∈X_(s). By using the term differentiable, it is assumed thatderivative of a function in point x exists and is bounded. At a minimum,there must be a first derivative where notation Y′(

(⋅)) denotes derivative y′(⋅) at each point in

(+), ∥⋅∥ is the l_(p) norm, τ^(t) is the threshold limiting the firstderivative, X₀≡Ø is an empty set, and respectively Y_(s)=Y(X_(s)). Itwill be understood that the common derivative function may be:y′(x ₁ ,x ₂)=y(x ₁+1,x ₂)−y(x ₁ ,x ₂)  (15)calculated in horizontal direction, and similarly can be calculated invertical direction, and that the function applies for both equations(13) and (14) (both the reference block and candidate block). Othertypes of derivative calculations that may be used include, Sobel,Prewitt, Laplacian, or other operators. The threshold τ¹ is generated byusing the same or similar process as that explained above for equation12. Here, it is assumed that the system is robust when an edge is lessthan 5, or 20, and so forth, and the threshold is set at 5*sqrt(K). Thethreshold is then refined with testing.

The use of the derivative may be very efficient because AF statisticsalready provides a good approximation for y′(⋅). Hence, there may be noneed to calculate the derivatives independently, and the derivatives maybe obtained from the AF statistics. Also, l₂ norm for the derivativecalculations is:∥α∥=(Σ_(i=1) ^(n)|α_(i)|²)^(1/2)  (16)similarly to l₂ distance d(⋅,⋅), while, it is worth to note, any normmay be equivalently efficient.

By another option, the block matching, and in turn statistics grid, maybe based on color values, and particularly color ratios. Specifically,in the case when color ratios are used instead of luminance values,conditions for bock similarity are as follows in Eqs. (17) to (19):

$\left\{ \begin{matrix}{{{x_{ref}\epsilon\; X} - {\bigcup_{i = 1}^{s - 1}X_{i}}},} & (17) \\{{{d\left( {{Y_{RG}\left( {\mathcal{B}\left( x_{ref} \right)} \right)},{Y_{RG}\left( {\mathcal{B}(x)} \right)}} \right)} \leq \tau},} & (18) \\{{{d\left( {{Y_{BG}\left( {\mathcal{B}\left( x_{ref} \right)} \right)},{Y_{BG}\left( {\mathcal{B}(x)} \right)}} \right)} \leq \tau},} & (19)\end{matrix} \right.$where equation (17) is similar to equation (11) above, and x≠x_(ref).Equations (18) and (19) are the Euclidean distances using color valueratios from the statistics grid or image data, such as:

$\begin{matrix}{{{Y_{RG}\left( {B( \cdot )} \right)} = \left\{ {\frac{y_{R}(x)}{y_{G}(x)}:{x\;\epsilon\;{B( \cdot )}}} \right\}},{and}} & (20) \\{{Y_{BG}\left( {B( \cdot )} \right)} = \left\{ {\frac{y_{R}(x)}{y_{G}(x)}:{x\;\epsilon\;{B( \cdot )}}} \right\}} & (21)\end{matrix}$where the ratio Y_(RG) denotes red y_(R) over green y_(G), and ratioY_(BG) is blue y_(B) over green y_(G) color ratios of input statistics,respectively. The thresholds for equations (18) and (19) are generatedas explained above for luminance distance equation (12).

For the best performance of self-similarity criteria from equations (11)to (14) and (17) to (19), luminance values y and color ratios

$\frac{y_{R}}{y_{G}},\frac{y_{B}}{y_{G}}$may be calculated from pre-processed statistics, which by one examplemay be ISP statistics, where shading is corrected by a correction grid(or statistics grid also called a LSC correction table herein) stored inmemory corresponding to a previous frame in a video stream, for example.In other words, while this system is operating for a video sequence withmultiple frames, there will be a correction grid from a previous frameas well as multiple candidate correction grids for the current frame.Thus, the option exists to use the previous frame correction grid sinceillumination conditions should not usually change that quickly fromsingle frame to single frame. It should be noted that whilepre-processed image data with LSC correction may be used to calculateblock similarities, the LSC correction tables (or correction grids)should be applied to raw image data only with black-level correction inorder to perform the shading correction on the image data.

Each block Y(

(x_(ref))) (e.g.

(⋅) of size 3×3 for example) is compared against other blocks of data Y(

(x)) with respect to a metric (Euclidean distance) d(⋅,⋅), as well astested

against the other criteria. If a result of the Euclidean metric iswithin a specified threshold, and all of the criteria mentioned above byone example are met, then two blocks are considered to be similar toeach other, and a match is formed (942). In this case, at least supportsor anchors of each block (sets of x coordinates where they are defined)are combined into a single ROI X_(s), or in other words, the matchedblock B(x) is added to the current cluster (or ROI X_(s)) (944). Thenext block is found (B(x)=B(x)+1) (946), and it is tested whether it ispast the last block position on the image (whether B(x)>B(N−K₁, M−K₂)(948). If not, the process 900 then returns to operation 938 to test tosee if the new block position B(x) already is within an ROI, and if sofinds the next block B(x) that is not in an ROI. Either way, thecomparison loop between B(X_(ref)) and the new B(x) is then performed asdescribed above. Alternatively, if the last block position has beenpassed, all of the available candidate block positions have already beencompared to the reference block B(X_(ref)), and process 900 continueswith an operation 950 to determine whether other minimum criteria forthe ROI Xs, such as size criteria described above, have been met. Ifnot, the present, new cluster or ROIXs just assembled is dropped, andthe process continues with operation 928 to select a new reference pointX_(ref) to start the matching loop over again. If the size criteria hasbeen met, total S is ticked up by one (S=S+1) (952), and the new ROI Xsis added to the ROI list for the current image (954). The process thenreturns to operation 926 or 928 to choose a new Xref position andpreform the matching loop again. This block-matching procedure may becarried out for each location:x _(ref) ∈{x=(x ₁ ,x ₂): x ₁=1, . . . ,N−K ₁ ,x ₂=1, . . . ,M−K ₂}  (22)

It will be understood that there may be many variations within the blockmatching process, and by one approach, as long as blocks are matchedupon a criteria to form a cluster that does not have a limit of thelocation of the block within the image and relative to other blocks inthe image. Also, a match may require some of the criteria, rather thanall of the listed criteria, for either the luminance or color ratiooptions. Further, both options may be provided alternatively andselected based on a determination as to which option will provide abetter result, or both options are performed and the best result is thenused. Also, other color spaces may be used such that u and v componentscan be used after YUV transformation, or a and b components can be usedafter Lab transformation.

The output from the block matching operation 812 that includes theprocess 900, includes:clusters of data Y _(s)={(y _(R)(x),y _(G1)(x),y _(G2)(x),y _(B)(x)):x∈X_(s) ,s=1, . . . ,S}  (23)or in other words, identification of the cluster (or ROI Xs), thelocation of the blocks in each cluster, and the luminance, and/or theBayer color values at the blocks within the cluster. It will beunderstood that with the identification of the block, the values relatedto that block may be looked-up in other locations on a memory, or thevalues may be collected onto a correction grid for fast retrieval duringthe LSC correction process.

It will be appreciated that there may be variations for the blockselection and cluster generation than that provided by process 900,where such a process may not have all of the operations of process 900,and/or the order of the operations may be different.

Returning to the process 800, process 800 may include “Revert image dataif needed” 813, and particularly, when the DS unit, or other unitproviding the image data to the 3A unit, can only provide the LSCcorrected image data, then that image data may be reverted back tonon-LSC corrected image data, such as into black-level-only correctedraw image data. As an example, if shading correction for red colorchannel is ŷ_(R) (x)−y_(R) (x)T^(R) (x) for all x ∈X, where y_(R) (⋅) isblack-level corrected input data and T^(R)(⋅) is the correctingfunction, then reverted correction for the red color channel isy_(R)(x)=ŷ_(R)(x)/T^(R)(x) for all x∈X. As mentioned above, thereversion may be performed by the DS unit or other module. Process 800then may include “apply selected candidate tables (Light 1 . . . , LightL) to the image data” 814, and by one example by an initial LSC unit624. As mentioned, each illuminant (or light source) detected aspossibly present for an image may have its own LSC table. Also, thecandidate tables are applied to non-LSC corrected (or black-levelcorrected) image data.

Process 800 may then include “perform AWB adjustments” 816, andparticularly to at least the image data within the defined clusters andwith the initial LSC correction tables applied. By one approach,grey-world (GW-AWB) algorithms are applied, and process 800 then mayinclude “convert RGB data to HSV color space” 818, which generates hueand saturation values for each of the points in the blocks in theclusters. Such GW-AWB corrections may be performed by the AWB module 627and controlled by the AWB unit 614.

Process 800 may include “calculate an error for a cluster perilluminant” 820. Specifically, a respective set of error values may bedetermined for each of the set of LSC tables. For example, LSCcorrection unit 612 may be operative to determine a respective set oferror values for each of a set of LSC tables 622 selected for theinitial LSC table unit 624. In some implementations, each set of errorvalues may describe error associated with application of itscorresponding LSC table to the black-level corrected sensor image. Invarious implementations, the pre-processed image may be segmented intomultiple clusters S of blocks as described above, and the set of errorvalues for each LSC table may comprise a set of cluster error values,each block error value comprising an error value for one of the multipleclusters.

The error unit 630 may calculate one or more errors e_(l,s) for eachilluminant/in a cluster-wise manner for each cluster s that is a resultof the block matching (or segmentation). The error unit 630 calculateserrors e_(l,s) in the HSV color space (e.g., via RGB to HSV unit 628)after application of a GW-AWB algorithm (e.g., via GW-AWB unit 627) withrespect to origin of coordinates (0 hue and 0 saturation) using equation(24) as follows:

$\begin{matrix}{e_{l,s} = {\frac{1}{A_{s}}{\sqrt{\sum_{i}{h_{i}^{2}s_{i}^{2}}}.}}} & (24)\end{matrix}$where here h is hue, s is saturation, A_(s) is cluster size (as in thearea of the cluster) measured by the number of pixels within the clusteror in other words #Xs. The l is an illuminant index. Value coordinatesare ignored.

In various implementations, the reliabilities of errors e_(l,s) may varyacross the various clusters s of an image. For example, in someimplementations, some clusters may be more affected by noise than otherclusters. Such non-uniformity may affect and tend to be especiallyprevalent under certain conditions, such as low-light conditions, forexample. The implementations are not limited in this context.

Process 800 also may include “calculate confidence value” 822, and byone example, performed by a probability and confidence unit 632. Aconfidence value c_(l,s) (see equations 32 and 33 below) may begenerated for each cluster s and for each illuminant 1. Specifically, alarger weight is given to an illuminant having a distributionconcentrated around 0, therefore giving a smaller error with respect toorigin, based on following probability equation:

$\begin{matrix}{P_{l,s} = \left\{ \begin{matrix}{1,{e_{l,s} \leq e_{\min}},} \\{\frac{e_{\max} - e_{l}}{e_{\max} - e_{\min}},{e_{\min} < e_{l} < e_{\max}}} \\{0,{otherwise},}\end{matrix} \right.} & (25)\end{matrix}$where e_(min)=min_(l){e_(l,s)}, e_(max)=e_(l,s)*tolerance, s is thecluster index, and tolerance is a relative constant representing howmuch error can be tolerated (e.g., a tolerance of 5% equals 1.05), orsimilar function not necessarily piece-wise linear, if computationalresources allow.

Any particular image analyzed for lens shading color correction may bedescribed as follows:z(x)=k(x)y(x)+σ(k(x)y(x)+b)n+b,  (26)where x=(x₁, x_(j)) is a spatial coordinate on the image plane, y(x) isa true expected unknown image value at spatial coordinate x, k is atwo-dimensional grid of attenuation coefficients, n˜N(0,1) is a randomwhite Gaussian noise of zero mean and unity standard deviation, b is theblack level constant, and a is a function of standard deviation fornoise, such that std{z(x)}=σ(y(x)).

In order to obtain a best estimate of the true image values in asub-region, a solution is sought in this form:{tilde over (y)}(x)=F{k ⁻¹(x)(z(x)−b)}  (27)where F{⋅} is an operator that indicates both grey-world white-balanceand color space transformation.

In order to account for varying reliabilities of values of y(x)calculated for various spatial locations, reliability parameters C(x)may be defined. In various implementations, values of C(x) may bedetermined based on values of Std(x), which may be calculated accordingto:Std(x)=std{H{{tilde over (y)}(x)}S{{tilde over (y)}(x)}}  (28)where H{⋅} indicates hue values, S{⋅} indicates saturation values, andstd represents the standard deviation operator. In some cases, y(x) maybe highly non-linear, and thus formal calculated of Std(x) may becomputationally intensive. In some implementations, in order to conservecomputational resources, C(x) may be defined using fixed approximationsof expected values for Std (x). For example, in various implementationsin which an image is divided into four rows and six columns ofsub-clusters, a set C of reliability parameters C(x) may be used thattakes the matrix form:

$\begin{matrix}{C = {\left\{ {\hat{c}}_{i,j}^{- 1} \right\} = \begin{bmatrix}256 & 4 & 2 & 2 & 4 & 256 \\16 & 2 & 1 & 1 & 2 & 16 \\16 & 2 & 1 & 1 & 2 & 16 \\256 & 4 & 2 & 2 & 4 & 256\end{bmatrix}}} & (29)\end{matrix}$where confidence c_(l,s) in color deviation estimate e_(l,s) in eachX_(s) is an important parameter that depends on spatial locationconfidence coefficient ĉ_(i,j) in each point x∈X_(s) (see FIG. 10 ). Theconfidence coefficient ĉ_(i,j) ⁻¹ represents the reciprocal of thecoefficient to be used for the sub-cluster positioned in the i^(th) rowand j^(th) column. In various implementations, the coefficients C_(l,s)in equation (30) may be weighted using c-J. In some implementations, theapproximations used in such a set of reliability parameters may beempirically-based. For example, in various implementations, suchapproximations may be based on simulation results and/or tests performedon actual image sensor data. The implementations are not limited in thiscontext. In some implementations, interpolation may be used for somepoints x in order to avoid abrupt changes in weighting when imageframing is only slightly changed. It is to be understood that in variousimplementations, C may differ from the example in Equation (29),depending on the applicable attenuation factors and/or noise model. Insome implementations, cluster sizes and shapes that are subject tosegmentation may vary, but reliability parameters C(x), similar to thatillustrated in Equation (29), may be fixed and stored for a wide rangesof sensors and/or modules. The implementations are not limited in thiscontext.

Thus, the matrix C(x) is a deviation of distribution of hue andsaturation calculated as sample quadratic error e_(l,s) with respect toorigin as in equation 26 within X_(s) for each candidate correction gridT_(l)(x). A larger amount of data points in X_(s) results in a higherconfidence c_(l,s):

$\begin{matrix}{{c_{l,s} = {\frac{1}{NM}\# X_{s}{\mathcal{p}}_{l,s}{\sum\limits_{x\;\epsilon\; X_{s}}{c(x)}}}},} & (30)\end{matrix}$where c(x) is interpolated from ĉ_(i,j) value in Eq. (31) at point x∈X,and p_(l,s) is a function of T_(l)(x) as in equations (33) to (35) belowand within X_(s).

Process 800 then may include “generate weights for LSC tables” 824, andparticularly to determine a weight (wl to wL as on FIG. 6 ) forindividual selected candidate correction tables based on the weights andthe confidence values. The weight calculations may be handled by theerror unit 630 or the fusion unit 634 that will also generate the finalcomposite LSC table, or it may be calculated by a separate weightgeneration unit. A respective weight may be determined for each of theset of LSC tables based on the corresponding set of error values forthat LSC table. For example, a respective weight may be determined foreach of a set of LSC tables shown at the LSC unit 624 and based oncorresponding sets of error values for each of those LSC tables. In someimplementations, a respective set of cluster weights may be calculatedfor each LSC table in the set, based on a corresponding set of clustererror values for that LSC table, and each cluster weight may correspondto one of the multiple clusters. In various implementations, therespective weight for each LSC table in the set may then be determinedas a weighted sum of the cluster weights for that LSC table. In somesuch implementations, a set of reliability parameters may be identifiedfor each of the multiple clusters, and the weighted sum of the set ofcluster weights for each LSC table may be calculated by weighting thecluster weights according to the reliability parameters for theircorresponding clusters. In various implementations, the reliabilityparameters may indicate levels of reliability for error values of theircorresponding clusters. The implementations are not limited in thiscontext.

Thus, individual table weights w₁ may be generated taking into accountconfidence c_(l,s) parameters of all clusters or ROIs X_(s), and colordistortion estimates e_(l,s) (for each candidate correction table T_(l))may be calculated as follows:

$\begin{matrix}{{\overset{\_}{w}}_{l} = {\sum\limits_{s = 1}^{S}{c_{l,s}e_{l,s}^{- 1}}}} & (31) \\{{\overset{\_}{w}}_{l} = \frac{{\overset{\_}{w}}_{l}}{\sum\limits_{i = 1}^{L}{\overset{\_}{w}}_{i}}} & (32)\end{matrix}$

Once the weight for each table w_(l) is generated for each LSC table,process 800 may include “generate composite LSC correction table” 826.The composite LSC table may be generated for the pre-processed imagebased on the respective weights for the set of LSC tables, and by oneexample the fusion unit 634. In some implementations, the composite LSCtable may be generated as a weighted sum of the set of LSC tables, basedon the respective weights.

Final correction table T(x) is a weighted sum of candidate correctiontables T_(l)(x):

$\begin{matrix}{{{T(x)} = {\sum\limits_{i = 1}^{L}{{T_{l}(x)}w_{l}}}},} & (33)\end{matrix}$The composite LSC table, or identification of the table, may be providedto the pre-processing unit 636 to be applied to the pre-processed imageto generate a processed image. This may or may not be via encoder 638 inorder to provide the image to a display device or compress the image forfurther transmission or for storage of the image. The implementationsare not limited to these examples.

Referring to FIG. 12 , process 1200 illustrates the operation of asample image processing system 1300 that performs lens shading colorcorrection in accordance with at least some implementations of thepresent disclosure. In more detail, in the illustrated form, process1200 may include one or more operations, functions or actions asillustrated by one or more of actions 1202 to 1216 numbered evenly. Byway of non-limiting example, process 1200 will be described herein withreference to FIG. 13 . Specifically, system 1300 includes logic units1304 that has a 3A unit 1310 with a lens shading color (LSC) estimationcomponent or unit 1312, and a block selection unit 1314. The operationof the system may proceed as follows.

Process 1200 may include “receive image data” 1202, and as explainedabove, it may be at least partially pre-processed image data, and it maybe in the form of a statistics grid of luminance or color values (orcolor value ratios). The block selection unit may receive image datathat is already LSC corrected in order to perform the block matchingmentioned below, but the lens shading color estimation unit shouldreceive non-LSC corrected image data for application of the candidateLSC tables. By one form, this latter image data is raw data that onlyreceived black-level correction so far.

Process 1200 may include “receive AF statistic derivative data” 1204,and particularly, point function derivatives when luminance values willbe used to determine whether blocks match for inclusion of the block ina cluster. Specifically, candidate blocks must match a reference blockfor inclusion.

Process 1200 may include “receive motion data” 1206 when motion data,such as motion vectors, will be used to at least initially locatereference points or blocks on a current image by using the location ofthe reference blocks from a previous image in order to reduce thecomputational load of block matching.

Process 1200 may include “generate ROI for current image based on motiondata”. Thus, as explained herein, the motion data may be used to locatethe reference points, and in turn the reference blocks, moved from aprevious image, to establish regions of interest (ROI) or clusters,based on the previous image. By one form, the motion established ROI areassumed valid, and by another form, the motion established ROI aretreated as preliminary where block matching comparisons are stillperformed to confirm the block membership of the motion-based ROI.

Process 1200 may include “generate new ROI based on search for referenceblock and matching blocks on area of image without ROI regardless of theposition of blocks on the image relative to each other” 1210.Particular, as described above, the system may search for new ROI inareas of the image that are not already covered by an ROI, whethermotion-based ROI or not by one example. Also, the inclusion in clustersmay be based on properties such as luminance and color values ratherthan any limit to which part of the image the blocks are located ortheir position relative to the position of other blocks. This refers tothe fact that criteria for the comparisons do not specifically mention ablock position or location on the image or relative to other blocks(other than not already being within an ROI).

Once the ROI are established for an image, the identification of the ROIand block membership within the ROI may be provided on a list of ROI forthe current image in a memory. This list or index is then made availablefor the lens shading color estimation unit. Ultimately, the process 1200then may “calculate errors using the ROIs”, and based on candidate LSCtables applied to raw image data that may be pre-processed byblack-level correction, and is then by at least basic WB correction andconversion to HSV color space as explained above. By one example, errorsare based on hue and saturation values as well as the area of thecluster or ROI, also as explained above with equation (24). Process 1200then may include “generate a weight for each candidate LSC table usingthe errors” 1214, and particularly to sum up weights from the individualclusters or ROI on an image to have a single weight for each candidateLSC table which may be provided for each illuminant. Process 1200 thanmay include “generate a composite table using the weights to apply tocorrect the color on an image” 1216, and particularly to determine asingle LSC correction table for the current image as described in detailabove. The correction table then may be applied to the pre-processedimage data before providing the image data to an encoder, storing thedata, transmitting the data, and/or displaying the image.

In addition, any one or more of the operations of FIGS. 1-12 may beundertaken in response to instructions provided by one or more computerprogram products. Such program products may include signal bearing mediaproviding instructions that, when executed by, for example, a processor,may provide the functionality described herein. The computer programproducts may be provided in any form of one or more machine-readablemedia. Thus, for example, a processor including one or more processorcore(s) may undertake one or more of the operations of the exampleprocesses herein in response to program code and/or instructions orinstruction sets conveyed to the processor by one or more computer ormachine-readable media. In general, a machine-readable medium may conveysoftware in the form of program code and/or instructions or instructionsets that may cause any of the devices and/or systems to perform asdescribed herein. The machine or computer readable media may be anon-transitory article or medium, such as a non-transitory computerreadable medium, and may be used with any of the examples mentionedabove or other examples except that it does not include a transitorysignal per se. It does include those elements other than a signal per sethat may hold data temporarily in a “transitory” fashion such as RAM andso forth.

As used in any implementation described herein, the term “module” refersto any combination of software logic, firmware logic and/or hardwarelogic configured to provide the functionality described herein. Thesoftware may be embodied as a software package, code and/or instructionset or instructions, and “hardware”, as used in any implementationdescribed herein, may include, for example, singly or in anycombination, hardwired circuitry, programmable circuitry, state machinecircuitry, and/or firmware that stores instructions executed byprogrammable circuitry. The modules may, collectively or individually,be embodied as circuitry that forms part of a larger system, forexample, an integrated circuit (IC), system on-chip (SoC), and so forth.For example, a module may be embodied in logic circuitry for theimplementation via software, firmware, or hardware of the coding systemsdiscussed herein.

As used in any implementation described herein, the term “logic unit”refers to any combination of firmware logic and/or hardware logicconfigured to provide the functionality described herein. The“hardware”, as used in any implementation described herein, may include,for example, singly or in any combination, hardwired circuitry,programmable circuitry, state machine circuitry, and/or firmware thatstores instructions executed by programmable circuitry. The logic unitsmay, collectively or individually, be embodied as circuitry that formspart of a larger system, for example, an integrated circuit (IC), systemon-chip (SoC), and so forth. For example, a logic unit may be embodiedin logic circuitry for the implementation firmware or hardware of thecoding systems discussed herein. One of ordinary skill in the art willappreciate that operations performed by hardware and/or firmware mayalternatively be implemented via software, which may be embodied as asoftware package, code and/or instruction set or instructions, and alsoappreciate that logic unit may also utilize a portion of software toimplement its functionality.

As used in any implementation described herein, the term “component” mayrefer to a module or to a logic unit, as these terms are describedabove. Accordingly, the term “component” may refer to any combination ofsoftware logic, firmware logic, and/or hardware logic configured toprovide the functionality described herein. For example, one of ordinaryskill in the art will appreciate that operations performed by hardwareand/or firmware may alternatively be implemented via a software module,which may be embodied as a software package, code and/or instructionset, and also appreciate that a logic unit may also utilize a portion ofsoftware to implement its functionality.

Referring to FIG. 13 , an example image processing system 1300 isarranged in accordance with at least some implementations of the presentdisclosure. In various implementations, the example image processingsystem 1300 may have an imaging device 1302 to form or receive capturedimage data. This can be implemented in various ways. Thus, in one form,the image processing system 1300 may be a digital camera or other imagecapture device, and imaging device 1302, in this case, may be the camerahardware and camera sensor software, module, or component. In otherexamples, imaging processing system 1300 may have an imaging device 1302that includes or may be a camera, and logic modules 1304 may communicateremotely with, or otherwise may be communicatively coupled to, theimaging device 1302 for further processing of the image data.

In either case, such technology may include a camera such as a digitalcamera system, a dedicated camera device, or an imaging phone, whether astill picture or video camera or some combination of both. Thus, in oneform, imaging device 1302 may include camera hardware and opticsincluding one or more sensors as well as auto-focus, zoom, aperture,ND-filter, auto-exposure, flash, and actuator controls. These controlsmay be part of a sensor module or component 1306 for operating thesensor. The sensor component 1306 may be part of the imaging device1302, or may be part of the logical modules 1304 or both. Such sensorcomponent can be used to generate images for a viewfinder and take stillpictures or video. The imaging device 1302 also may have a lens, animage sensor with a RGB Bayer color filter, an analog amplifier, an A/Dconverter, other components to convert incident light into a digitalsignal, the like, and/or combinations thereof. The digital signal alsomay be referred to as the raw image data herein.

Other forms include a camera sensor-type imaging device or the like (forexample, a webcam or webcam sensor or other complementarymetal-oxide-semiconductor-type image sensor (CMOS)), without the use ofa red-green-blue (RGB) depth camera and/or microphone-array to locatewho is speaking. The camera sensor may also support other types ofelectronic shutters, such as global shutter in addition to, or insteadof, rolling shutter, and many other shutter types. In other examples, anRGB-Depth camera and/or microphone-array might be used in addition to orin the alternative to a camera sensor. In some examples, imaging device1302 may be provided with an eye tracking camera. The imaging device1302 also may have a lens actuator or control 1308 that has a lens and alens driver or driver logic that receives commands to move the lens andapplies an electrical current to move the lens.

In the illustrated example, the logic modules 1304 may include the 3Acomponent that, in turn, may include an AF component 1318 to compute adesired new current lens position, an automatic white balance (AWB)component 1316, and an automatic exposure control (AEC) component 1317.The 3A component unit 1310 also may have a lens shading color (LSC)correction/table component 1312 which in turn may have a block selectionunit 1314 that performs many of the operations described herein. The 3Acomponent unit 1310 may be operated by, or even entirely or partiallylocated at, processor(s) 1320, and which may include an ISP 1322 toperform the operations. The logic modules may be communicatively coupledto the components of the imaging device 1302 in order to receive rawimage data. In these cases, it is assumed the logic modules 1304 areconsidered to be separate from the imaging device. This need not be so,and the logic modules very well may be considered to be part of theimaging device as well.

The image processing system 1300 may have one or more processors 1320which may include a dedicated image signal processor (ISP) 1322 such asthe Intel Atom, memory stores 1324 which may or may not hold the LSCtables and other LSC correction data, one or more displays 1328 toprovide images 1330, encoder 1332 that has an inter-prediction unit 1336to provide motion data such as motion vectors for example, and antenna1334. In one example implementation, the image processing system 1300may have the display 1328, at least one processor 1320 communicativelycoupled to the display, at least one memory 1324 communicatively coupledto the processor to perform the operations described herein as explainedabove. The encoder 1332 and antenna 1334 may be provided to compress themodified image date for transmission to other devices that may displayor store the image as well as provide the motion data for blockmatching. It will be understood that the image processing system 1300may also include a decoder (or encoder 1332 may include a decoder) toreceive and decode image data for processing by the system 1300.Otherwise, the processed image 1330 may be displayed on display 1328 orstored in memory 1324. As illustrated, any of these components may becapable of communication with one another and/or communication withportions of logic modules 1304 and/or imaging device 1302. Thus,processors 1320 may be communicatively coupled to both the image device1302 and the logic modules 1304 for operating those components. By oneapproach, although image processing system 1300, as shown in FIG. 13 ,may include one particular set of blocks or actions associated withparticular components or modules, these blocks or actions may beassociated with different components or modules than the particularcomponent or module illustrated here.

Referring to FIG. 14 , an example system 1400 in accordance with thepresent disclosure operates one or more aspects of the image processingsystem described herein. It will be understood from the nature of thesystem components described below that such components may be associatedwith, or used to operate, certain part or parts of the image processingsystem described above. In various implementations, system 1400 may be amedia system although system 1400 is not limited to this context. Forexample, system 1400 may be incorporated into a digital still camera,digital video camera, mobile device with camera or video functions suchas an imaging phone, webcam, personal computer (PC), laptop computer,ultra-laptop computer, tablet, touch pad, portable computer, handheldcomputer, palmtop computer, personal digital assistant (PDA), cellulartelephone, combination cellular telephone/PDA, television, smart device(e.g., smart phone, smart tablet or smart television), mobile internetdevice (MID), messaging device, data communication device, and so forth.

In various implementations, system 1400 includes a platform 1402 coupledto a display 1420. Platform 1402 may receive content from a contentdevice such as content services device(s) 1430 or content deliverydevice(s) 1440 or other similar content sources. A navigation controller1450 including one or more navigation features may be used to interactwith, for example, platform 1402 and/or display 1420. Each of thesecomponents is described in greater detail below.

In various implementations, platform 1402 may include any combination ofa chipset 1405, processor 1410, memory 1412, storage 1414, graphicssubsystem 1415, applications 1416 and/or radio 1418. Chipset 1405 mayprovide intercommunication among processor 1410, memory 1412, storage1414, graphics subsystem 1415, applications 1416 and/or radio 1418. Forexample, chipset 1405 may include a storage adapter (not depicted)capable of providing intercommunication with storage 1414.

Processor 1410 may be implemented as a Complex Instruction Set Computer(CISC) or Reduced Instruction Set Computer (RISC) processors; x86instruction set compatible processors, multi-core, or any othermicroprocessor or central processing unit (CPU). In variousimplementations, processor 1410 may be dual-core processor(s), dual-coremobile processor(s), and so forth.

Memory 1412 may be implemented as a volatile memory device such as, butnot limited to, a Random Access Memory (RAM), Dynamic Random AccessMemory (DRAM), or Static RAM (SRAM).

Storage 1414 may be implemented as a non-volatile storage device suchas, but not limited to, a magnetic disk drive, optical disk drive, tapedrive, an internal storage device, an attached storage device, flashmemory, battery backed-up SDRAM (synchronous DRAM), and/or a networkaccessible storage device. In various implementations, storage 1414 mayinclude technology to increase the storage performance enhancedprotection for valuable digital media when multiple hard drives areincluded, for example.

Graphics subsystem 1415 may perform processing of images such as stillor video for display. Graphics subsystem 1415 may be a graphicsprocessing unit (GPU) or a visual processing unit (VPU), for example. Ananalog or digital interface may be used to communicatively couplegraphics subsystem 1415 and display 1420. For example, the interface maybe any of a High-Definition Multimedia Interface, Display Port, wirelessHDMI, and/or wireless HD compliant techniques. Graphics subsystem 1415may be integrated into processor 1410 or chipset 1405. In someimplementations, graphics subsystem 1415 may be a stand-alone cardcommunicatively coupled to chipset 1405.

The graphics and/or video processing techniques described herein may beimplemented in various hardware architectures. For example, graphicsand/or video functionality may be integrated within a chipset.Alternatively, a discrete graphics and/or video processor may be used.As still another implementation, the graphics and/or video functions maybe provided by a general purpose processor, including a multi-coreprocessor. In further implementations, the functions may be implementedin a consumer electronics device.

Radio 1418 may include one or more radios capable of transmitting andreceiving signals using various suitable wireless communicationstechniques. Such techniques may involve communications across one ormore wireless networks. Example wireless networks include (but are notlimited to) wireless local area networks (WLANs), wireless personal areanetworks (WPANs), wireless metropolitan area network (WMANs), cellularnetworks, and satellite networks. In communicating across such networks,radio 1418 may operate in accordance with one or more applicablestandards in any version.

In various implementations, display 1420 may include any television typemonitor or display. Display 1420 may include, for example, a computerdisplay screen, touch screen display, video monitor, television-likedevice, and/or a television. Display 1420 may be digital and/or analog.In various implementations, display 1420 may be a holographic display.Also, display 1420 may be a transparent surface that may receive avisual projection. Such projections may convey various forms ofinformation, images, and/or objects. For example, such projections maybe a visual overlay for a mobile augmented reality (MAR) application.Under the control of one or more software applications 1416, platform1402 may display user interface 1422 on display 1420.

In various implementations, content services device(s) 1430 may behosted by any national, international and/or independent service andthus accessible to platform 1402 via the Internet, for example. Contentservices device(s) 1430 may be coupled to platform 1402 and/or todisplay 1420. Platform 1402 and/or content services device(s) 1430 maybe coupled to a network 1460 to communicate (e.g., send and/or receive)media information to and from network 1460. Content delivery device(s)1440 also may be coupled to platform 1402 and/or to display 1420.

In various implementations, content services device(s) 1430 may includea cable television box, personal computer, network, telephone, Internetenabled devices or appliance capable of delivering digital informationand/or content, and any other similar device capable of unidirectionallyor bidirectionally communicating content between content providers andplatform 1402 and/display 1420, via network 1460 or directly. It will beappreciated that the content may be communicated unidirectionally and/orbidirectionally to and from any one of the components in system 1400 anda content provider via network 1460. Examples of content may include anymedia information including, for example, video, music, medical andgaming information, and so forth.

Content services device(s) 1430 may receive content such as cabletelevision programming including media information, digital information,and/or other content. Examples of content providers may include anycable or satellite television or radio or Internet content providers.The provided examples are not meant to limit implementations inaccordance with the present disclosure in any way.

In various implementations, platform 1402 may receive control signalsfrom navigation controller 1450 having one or more navigation features.The navigation features of controller 1450 may be used to interact withuser interface 1422, for example. In implementations, navigationcontroller 1450 may be a pointing device that may be a computer hardwarecomponent (specifically, a human interface device) that allows a user toinput spatial (e.g., continuous and multi-dimensional) data into acomputer. Many systems such as graphical user interfaces (GUI), andtelevisions and monitors allow the user to control and provide data tothe computer or television using physical gestures.

Movements of the navigation features of controller 1450 may bereplicated on a display (e.g., display 1420) by movements of a pointer,cursor, focus ring, or other visual indicators displayed on the display.For example, under the control of software applications 1416, thenavigation features located on navigation controller 1450 may be mappedto virtual navigation features displayed on user interface 1422, forexample. In implementations, controller 1450 may not be a separatecomponent but may be integrated into platform 1402 and/or display 1420.The present disclosure, however, is not limited to the elements or inthe context shown or described herein.

In various implementations, drivers (not shown) may include technologyto enable users to instantly turn on and off platform 1402 like atelevision with the touch of a button after initial boot-up, whenenabled, for example. Program logic may allow platform 1402 to streamcontent to media adaptors or other content services device(s) 1430 orcontent delivery device(s) 1440 even when the platform is turned “off.”In addition, chipset 1405 may include hardware and/or software supportfor 8.1 surround sound audio and/or high definition (7.1) surround soundaudio, for example. Drivers may include a graphics driver for integratedgraphics platforms. In implementations, the graphics driver may comprisea peripheral component interconnect (PCI) Express graphics card.

In various implementations, any one or more of the components shown insystem 1400 may be integrated. For example, platform 1402 and contentservices device(s) 1430 may be integrated, or platform 1402 and contentdelivery device(s) 1440 may be integrated, or platform 1402, contentservices device(s) 1430, and content delivery device(s) 1440 may beintegrated, for example. In various implementations, platform 1402 anddisplay 1420 may be an integrated unit. Display 1420 and content servicedevice(s) 1430 may be integrated, or display 1420 and content deliverydevice(s) 1440 may be integrated, for example. These examples are notmeant to limit the present disclosure.

In various implementations, system 1400 may be implemented as a wirelesssystem, a wired system, or a combination of both. When implemented as awireless system, system 1400 may include components and interfacessuitable for communicating over a wireless shared media, such as one ormore antennas, transmitters, receivers, transceivers, amplifiers,filters, control logic, and so forth. An example of wireless sharedmedia may include portions of a wireless spectrum, such as the RFspectrum and so forth. When implemented as a wired system, system 1900may include components and interfaces suitable for communicating overwired communications media, such as input/output (I/O) adapters,physical connectors to connect the I/O adapter with a correspondingwired communications medium, a network interface card (NIC), disccontroller, video controller, audio controller, and the like. Examplesof wired communications media may include a wire, cable, metal leads,printed circuit board (PCB), backplane, switch fabric, semiconductormaterial, twisted-pair wire, co-axial cable, fiber optics, and so forth.

Platform 1402 may establish one or more logical or physical channels tocommunicate information. The information may include media informationand control information. Media information may refer to any datarepresenting content meant for a user. Examples of content may include,for example, data from a voice conversation, videoconference, streamingvideo, electronic mail (“email”) message, voice mail message,alphanumeric symbols, graphics, image, video, text and so forth. Datafrom a voice conversation may be, for example, speech information,silence periods, background noise, comfort noise, tones and so forth.Control information may refer to any data representing commands,instructions or control words meant for an automated system. Forexample, control information may be used to route media informationthrough a system, or instruct a node to process the media information ina predetermined manner. The implementations, however, are not limited tothe elements or in the context shown or described in FIG. 14 .

Referring to FIG. 15 , a small form factor device 1500 is one example ofthe varying physical styles or form factors in which system 1400 may beembodied. By this approach, device 1500 may be implemented as a mobilecomputing device having wireless capabilities. A mobile computing devicemay refer to any device having a processing system and a mobile powersource or supply, such as one or more batteries, for example.

As described above, examples of a mobile computing device may include adigital still camera, digital video camera, mobile devices with cameraor video functions such as imaging phones, webcam, personal computer(PC), laptop computer, ultra-laptop computer, tablet, touch pad,portable computer, handheld computer, palmtop computer, personal digitalassistant (PDA), cellular telephone, combination cellular telephone/PDA,television, smart device (e.g., smart phone, smart tablet or smarttelevision), mobile internet device (MID), messaging device, datacommunication device, and so forth.

Examples of a mobile computing device also may include computers thatare arranged to be worn by a person, such as a wrist computer, fingercomputer, ring computer, eyeglass computer, belt-clip computer, arm-bandcomputer, shoe computers, clothing computers, and other wearablecomputers. In various implementations, for example, a mobile computingdevice may be implemented as a smart phone capable of executing computerapplications, as well as voice communications and/or datacommunications. Although some implementations may be described with amobile computing device implemented as a smart phone by way of example,it may be appreciated that other implementations may be implementedusing other wireless mobile computing devices as well. Theimplementations are not limited in this context.

As shown in FIG. 15 , device 1500 may include a housing 1502, a display1504 including a screen 1510, an input/output (I/O) device 1506, and anantenna 1508. Device 1500 also may include navigation features 1512.Display 1504 may include any suitable display unit for displayinginformation appropriate for a mobile computing device. I/O device 1506may include any suitable I/O device for entering information into amobile computing device. Examples for I/O device 1506 may include analphanumeric keyboard, a numeric keypad, a touch pad, input keys,buttons, switches, rocker switches, microphones, speakers, voicerecognition device and software, and so forth. Information also may beentered into device 1500 by way of microphone (not shown). Suchinformation may be digitized by a voice recognition device (not shown).The implementations are not limited in this context.

Various forms of the devices and processes described herein may beimplemented using hardware elements, software elements, or a combinationof both. Examples of hardware elements may include processors,microprocessors, circuits, circuit elements (e.g., transistors,resistors, capacitors, inductors, and so forth), integrated circuits,application specific integrated circuits (ASIC), programmable logicdevices (PLD), digital signal processors (DSP), field programmable gatearray (FPGA), logic gates, registers, semiconductor device, chips,microchips, chip sets, and so forth. Examples of software may includesoftware components, programs, applications, computer programs,application programs, system programs, machine programs, operatingsystem software, middleware, firmware, software modules, routines,subroutines, functions, methods, procedures, software interfaces,application program interfaces (API), instruction sets, computing code,computer code, code segments, computer code segments, words, values,symbols, or any combination thereof. Determining whether animplementation is implemented using hardware elements and/or softwareelements may vary in accordance with any number of factors, such asdesired computational rate, power levels, heat tolerances, processingcycle budget, input data rates, output data rates, memory resources,data bus speeds and other design or performance constraints.

One or more aspects of at least one implementation may be implemented byrepresentative instructions stored on a machine-readable medium whichrepresents various logic within the processor, which when read by amachine causes the machine to fabricate logic to perform the techniquesdescribed herein. Such representations, known as “IP cores” may bestored on a tangible, machine readable medium and supplied to variouscustomers or manufacturing facilities to load into the fabricationmachines that actually make the logic or processor.

While certain features set forth herein have been described withreference to various implementations, this description is not intendedto be construed in a limiting sense. Hence, various modifications of theimplementations described herein, as well as other implementations,which are apparent to persons skilled in the art to which the presentdisclosure pertains are deemed to lie within the spirit and scope of thepresent disclosure.

The following examples pertain to further implementations.

By one example, a computer-implemented method of lens shading colorcorrection, comprises obtaining at least one statistics grid of pixeldata spatially corresponding to a frame having blocks, and selecting atleast some of the blocks to be included in at least one cluster ofblocks, and comprising choosing candidate blocks at least with adifference in luminance or color ratio between the candidate block and areference block that meets a criteria. The method also may comprisefinding an error associated with the color in the cluster; using theerrors to determine a weight for one or more LSC tables; and forming acomposite LSC table based on the weights and to be applied to correctcolor at the frame.

By another implementation, the method may comprise at least one of: (1)the criteria indicates a more constant color in an area on the frame,and wherein the criteria is a threshold of a maximum change in colorratio, and (2) the criteria indicates smoothness in an area of theframe, and wherein the criteria is a first threshold of a maximum changein luminance, and wherein a block is selected for inclusion in thecluster when the first threshold is met and when the luminance functionat the location of the block has a derivative that is less than a secondthreshold. Otherwise, the method may comprise obtaining the derivativesfrom autofocus statistics, wherein blocks are included in the clusterregardless of the position of the block on the frame and the positionrelative to the position of other blocks on the frame, wherein selectingcomprises selecting a reference block and comparing multiple blocks inthe frame to the reference block and that are not already consideredpart of any cluster, and wherein the difference is calculated as aEuclidean distance by differencing pixel values of corresponding pixelpositions within the two blocks being compared.

By other approaches, the method also may comprise providing colorcorrection for at least one frame in a video sequence of framescomprising using motion vectors and using the motion vectors to formclusters on the current image, wherein pixel data of blocks on the framethat are not within the cluster are not used to calculate color errorsfor lens shading color correction, wherein a reference point on thestatistics grid is selected by searching for a point row-by-row and thatis not already in a cluster, wherein the error and table weightcalculations are performed on a cluster-by-cluster basis rather than ablock-by-block basis, and wherein the luminance or color ratio imagedata used to compare candidate blocks to a reference block is obtainedfrom already lens-shading-color (LSC) corrected data, and whereinforming the final composite LSC table is based on NON-LSC correcteddata.

By yet another implementation, a computer-implemented system has of adisplay, at least one processor communicatively coupled to the display,at least one memory communicatively coupled to at least one processorand storing image capture data of at least one frame of a video sequenceor at least one still photograph, and at least one lens shadingcorrection (LSC) unit communicatively coupled to the processor. The atleast one lens shading correction unit is provided to obtain at leastone statistics grid of pixel data spatially corresponding to a framehaving blocks, select blocks to be included in at least one cluster ofthe blocks and comprising choosing candidate blocks at least with adifference in luminance or color ratio between the candidate block and areference block that meets a criteria, find an error associated with thecolor in the cluster, use the errors to determine a weight for one ormore LSC tables, and form a composite LSC table based on the weights andto be applied to correct color at the frame.

By another example, the system includes wherein at least one of: (1) thecriteria indicates a more constant color in an area on the frame, andwherein the criteria is a threshold of a maximum change in color ratio,and (2) the criteria indicates smoothness in an area of the frame, andwherein the criteria is a first threshold of a maximum change inluminance, and wherein a block is selected for inclusion in the clusterwhen the first threshold is met and when the luminance function at thelocation of the block has a derivative that is less than a secondthreshold. Otherwise, the system provides the at least one LSCcorrection unit to obtain the derivatives from autofocus statistics,wherein blocks are included in the cluster regardless of the position ofthe block on the frame and the position relative to the position ofother blocks on the frame, wherein selecting comprises selecting areference block and comparing multiple blocks in the frame to thereference block and that are not already considered part of any cluster,and wherein the difference is calculated as a Euclidean distance bydifferencing pixel values of corresponding pixel positions within thetwo blocks being compared.

By other approaches, the system provides the at least one LSC correctionunit to provide color correction for at least one frame in a videosequence of frames comprising using motion vectors and using the motionvectors to form clusters on the current image, wherein pixel data ofblocks on the frame that are not within the cluster are not used tocalculate color errors for lens shading color correction, wherein areference point on the statistics grid is selected by searching for apoint row-by-row and that is not already in a cluster, wherein the errorand table weight calculations are performed on a cluster-by-clusterbasis rather than a block-by-block basis, and wherein the luminance orcolor ratio image data used to compare candidate blocks to a referenceblock is obtained from already lens-shading-color (LSC) corrected data,and wherein forming the final composite LSC table is based on NON-LSCcorrected data.

By one approach, at least one computer readable medium comprises aplurality of instructions that in response to being executed on acomputing device, causes the computing device to: obtain at least onestatistics grid of pixel data spatially corresponding to a frame havingblocks, select blocks to be included in at least one cluster of theblocks and comprising choosing candidate blocks at least with adifference in luminance or color ratio between the candidate block and areference block that meets a criteria, find an error associated with thecolor in the cluster, use the errors to determine a weight for one ormore LSC tables, and form a composite LSC table based on the weights andto be applied to correct color at the frame.

By another approach, the instructions cause the computing device to useat least one of: (1) the criteria indicates a more constant color in anarea on the frame, and wherein the criteria is a threshold of a maximumchange in color ratio, and (2) the criteria indicates smoothness in anarea of the frame, and wherein the criteria is a first threshold of amaximum change in luminance, and wherein a block is selected forinclusion in the cluster when the first threshold is met and when theluminance function at the location of the block has a derivative that isless than a second threshold. Otherwise, the instructions cause thecomputing device to obtain the derivatives from autofocus statistics,wherein blocks are included in the cluster regardless of the position ofthe block on the frame and the position relative to the position ofother blocks on the frame, wherein selecting comprises selecting areference block and comparing multiple blocks in the frame to thereference block and that are not already considered part of any cluster,and wherein the difference is calculated as a Euclidean distance bydifferencing pixel values of corresponding pixel positions within thetwo blocks being compared.

By other approaches, the instructions cause the computing device toprovide color correction for at least one frame in a video sequence offrames comprising using motion vectors and using the motion vectors toform clusters on the current image, wherein pixel data of blocks on theframe that are not within the cluster are not used to calculate colorerrors for lens shading color correction, wherein a reference point onthe statistics grid is selected by searching for a point row-by-row andthat is not already in a cluster, wherein the error and table weightcalculations are performed on a cluster-by-cluster basis rather than ablock-by-block basis, wherein the luminance or color ratio image dataused to compare candidate blocks to a reference block is obtained fromalready lens-shading-color (LSC) corrected data, and wherein forming thefinal composite LSC table is based on NON-LSC corrected data.

In a further example, at least one machine readable medium may include aplurality of instructions that in response to being executed on acomputing device, causes the computing device to perform the methodaccording to any one of the above examples.

In a still further example, an apparatus may include means forperforming the methods according to any one of the above examples.

The above examples may include specific combination of features.However, the above examples are not limited in this regard and, invarious implementations, the above examples may include undertaking onlya subset of such features, undertaking a different order of suchfeatures, undertaking a different combination of such features, and/orundertaking additional features than those features explicitly listed.For example, all features described with respect to any example methodsherein may be implemented with respect to any example apparatus, examplesystems, and/or example articles, and vice versa.

What is claimed is:
 1. An apparatus comprising: memory; instructions;and processor circuitry to execute the instructions to: assign blocks ofpixels that have not been corrected for color shading to one or moreclusters based on one or more thresholds and color ratios associatedwith the blocks, the pixels associated with an image; determine at leastone modification parameter based on the one or more clusters; modifyfirst shading correction data based on the at least one modificationparameter to determine second shading correction data; and correct colorshading in the image based on the second shading correction data.
 2. Theapparatus of claim 1, wherein the at least one modification parameterincludes a scale value, and to modify the first shading correction data,the processor circuitry is to scale the first shade correction data withthe scale value to determine the second shade correction data.
 3. Theapparatus of claim 1, wherein the processor circuitry is to determinethe at least one modification parameter based on hue values associatedwith the one or more clusters.
 4. The apparatus of claim 1, wherein todetermine the at least one modification parameter, the processorcircuitry is to: determine an error value based on the one or moreclusters; and determine the at least one modification parameter based onthe error value.
 5. The apparatus of claim 1, wherein the color ratiosinclude at least one of red to green color ratios or blue to green colorratios.
 6. The apparatus of claim 1, wherein the color ratios includered to green color ratios and blue to green color ratios.
 7. At leastone non-transitory computer readable medium comprising computer readableinstructions that, when executed, cause processor circuitry to at least:assign blocks of pixels that have not been corrected for color shadingto one or more clusters based on one or more thresholds and color ratiosassociated with the blocks, the pixels associated with an image;determine at least one modification parameter based on the one or moreclusters; modify first shading correction data based on the at least onemodification parameter to determine second shading correction data; andcorrect color shading in the image based on the second shadingcorrection data.
 8. The at least one non-transitory computer readablemedium of claim 7, wherein the at least one modification parameterincludes a scale value, and to modify the first shading correction data,the instructions are to cause the processor circuitry to scale the firstshade correction data with the scale value to determine the second shadecorrection data.
 9. The at least one non-transitory computer readablemedium of claim 7, wherein the instructions are to cause the processorcircuitry to determine the at least one modification parameter based onhue values associated with the one or more clusters.
 10. The at leastone non-transitory computer readable medium of claim 7, wherein todetermine the at least one modification parameter, the instructions areto cause the processor circuitry to: determine an error value based onthe one or more clusters; and determine the at least one modificationparameter based on the error value.
 11. The at least one non-transitorycomputer readable medium of claim 7, wherein the color ratios include atleast one of red to green color ratios or blue to green color ratios.12. The at least one non-transitory computer readable medium of claim 7,wherein the color ratios include red to green color ratios and blue togreen color ratios.
 13. A method comprising: assigning, by executing aninstruction with processor circuitry, blocks of pixels that have notbeen corrected for color shading to one or more clusters based on one ormore thresholds and color ratios associated with the blocks, the pixelsassociated with an image; determining, by executing an instruction withprocessor circuitry, at least one modification parameter based on theone or more clusters; modifying, by executing an instruction withprocessor circuitry, first shading correction data based on the at leastone modification parameter to determine second shading correction data;and correcting color shading in the image based on the second shadingcorrection data.
 14. The method of claim 13, wherein the at least onemodification parameter includes a scale value, and the modifying of thefirst shading correction data includes scaling the first shadecorrection data with the scale value to determine the second shadecorrection data.
 15. The method of claim 13, wherein the determining ofthe at least one modification parameter is based on hue valuesassociated with the one or more clusters.
 16. The method of claim 13,wherein the determining of the at least one modification parameterincludes: determining an error value based on the one or more clusters;and determining the at least one modification parameter based on theerror value.
 17. The method of claim 13, wherein the color ratiosinclude at least one of red to green color ratios or blue to green colorratios.
 18. The method of claim 13, wherein the color ratios include redto green color ratios and blue to green color ratios.